The present disclosure relates to a an imaging device, provided with a plurality of imaging elements, an image processing method, and a computer program, and particularly to an imaging device, an image processing method, and a computer program, which associates a captured image obtained by the imaging elements.
A digital camera which digitally coding an image captured by an image sensor constituted by solid-state imaging elements such as a CCD (Charge Coupled Device), a CMOS (Complementary Metal Oxide Semiconductor), or the like has been widely distributed in stead of a silver salt camera which captures image with the use of a film or a photographic plate. A digital camera has advantages in that it is possible to store the digitally coded image on a memory and perform image processing and image management thereon by a computer and in that there is no concern of a life span of a film.
Some imaging elements have a function of detecting phase difference information. It is possible to detect a focus state of an imaging lens based on the phase difference in a pair of image signals output from a specific photoelectric conversion unit (photodiode).
Although a person who is not skilled in imaging intensely moves a camera due to the unfamiliarity with operations (for example, strong hand shaking) in many cases, many digital cameras mount an image stabilization function. The image stabilization function includes an electronic image stabilization for correction by image processing on the image sensor and mechanical image stabilization for correction with an optical lens.
The electronic image stabilization is a correcting method in which an effective pixel region extracted as a captured image from a maximum pixel region which can be captured is moved in accordance with the amount of image stabilization. According to the electronic image stabilization, although it is not necessary to mount a special mechanism, there is a disadvantage in that the light receiving area of the imaging elements is not effectively used.
On the other hand, the mechanical image stabilization function can be realized by sending feedback of a hand shaking vector measured with the use of a gyro sensor or an acceleration sensor to a mechanical system and performing high-speed control so as not to generate in the image projected to the image sensor. As the mechanical system described herein, a lens, a prism, and an imager (or a module integrated with an imager) are exemplified and respectively referred to as a “lens shift”, a “prism shift” and an “imager shift”. According to the image stabilization based on the lens shift scheme, it is necessary to embed a special mechanism which drives a specific lens, and the costs becomes higher since it becomes difficult to design the camera lens due to the increased size. On the other hand, the image stabilization based on the imager shift scheme, it is possible to perform the image stabilization with all camera lenses which can be installed on the camera main body, and it is not necessary to design a special camera lens.
Recently, many digital cameras are provided with a plurality of image capturing units. For example, a camera device which is provided with a branching unit which branches an optical path into a plurality of optical paths and a plurality of imaging elements disposed at an image forming position of each optical path, configured to output video signals of images with different image angle sizes from each of the imaging elements by differentiating the imaging sizes of the imaging elements, and capable of independently obtaining video signals of images with a plurality of image angles has been proposed (see Japanese Unexamined Patent Application Publication No. 2003-304115, for example).
In addition, an imaging device which is provided with a sub imaging element which captures the auxiliary captured image such as an object image to be displayed as a live view on the finder window as well as a main imaging element which captures a main captured image has been proposed (see Japanese Unexamined Patent Application Publication No. 2010-224112, for example). A user can check framing or the like while viewing the auxiliary captured image at the time of capturing an image.
It is expected to provide new applications for the digital camera provided with a plurality of imaging units by associating the captured image obtained by each imaging unit.
While the main imaging element has high pixels for capturing a stationary image, however, the sub imaging element has low pixels since the auxiliary image is captured mainly for a live view, namely a moving image. That is, it is difficult to associate both captured images since the screen sizes, the number of pixels, and pixel pitches are different in each imaging element. In addition, the image capturing timing does not completely coincide due to different frame rates resulting from the difference in the number of pixels of both imaging elements. Accordingly, if the hand shaking of the person, who captures an image, at the time of capturing image is faster than the timing of obtaining the image, it becomes difficult to take an action between the captured images in some cases.
In addition, a space in the camera main body, in which the aforementioned image stabilization mechanical system can be disposed, is limited, and the image stabilization mechanism based on the imager shift scheme is generally installed only on the side of the main imaging element. If image stabilization is performed on only one side of the imaging elements, it further becomes difficult to associate between captured images. In addition, it is difficult to associate between captured images even if the image stabilization mechanisms are installed in both imaging elements, due to a correction error, a computation error of a displacement amount of the imaging elements, and a displacement error.